Analysis of Shaw Brothers Kung Fu Films

Analysis and other experimentation for my Data Driven Shaw Brothers Analysis.

Follow along here!

library(scales)

Attaching package: ‘scales’

The following object is masked from ‘package:purrr’:

    discard

The following objects are masked from ‘package:readr’:

    col_factor, col_numeric

library(ggthemes)
library(tidyjson)
library(tidyverse)
library(forcats)
source("v_theme.R")

Read in Data

First, we need to get the data somehow. Its in JSON format, so let’s read the JSON into a string.

filename = '../out/shaw.json'
shaw_json <- paste(readLines(filename), collapse="")

incomplete final line found on '../out/shaw.json'

Then use tidyjson to convert the nested form into a flat data frame (tibble) we can work with.

films <- shaw_json %>% as.tbl_json %>% gather_array %>%
  spread_values(
    title = jstring("title"),
    director = jstring('director'),
    year = jstring('year'),
    avg_rating = jstring('avg_rating'),
    watches = jstring("watches"),
    likes = jstring("likes")
  )
films %>% head(n = 5) %>% select(title, director, year)

The above table has a row for every film.

We can use tidyjson again to create a row for each actor, duplicating the film-specific data for each actor that had a part in it.

cast <- shaw_json  %>% as.tbl_json  %>% gather_array %>%
  spread_values(
    title = jstring("title"),
    director = jstring('director'),
    year = jstring('year'),
    avg_rating = jstring('avg_rating'),
    watches = jstring("watches"),
    likes = jstring("likes")
  ) %>% enter_object("cast") %>% gather_array() %>%
  spread_values(
    name = jstring("name")
  )
cast %>% head(n = 8) %>% select(title, year, name)

Right now characters are a separte array

characters <- shaw_json  %>% as.tbl_json  %>% gather_array %>%
  spread_values(
    title = jstring("title"),
    director = jstring('director'),
    year = jstring('year'),
    avg_rating = jstring('avg_rating'),
    watches = jstring("watches"),
    likes = jstring("likes")
  ) %>% enter_object("characters") %>% gather_array() %>%
  spread_values(
    name = jstring("name")
  )

Real quick, let’s get a sense of the number of films in our data.

nrow(films)

[1] 260

260! That’s a lot of Kung-Fu. Let’s take a look at these films from a few different angles. We can start with release year.

Shaw Brothers, Through The Ages

So, I said retro, when exactly were these movies made?

source("v_theme.R")
films %>% ggplot(aes(x = year)) +
  geom_bar() +
  labs(title = 'Shaw Bros Films by Year') + 
  #fte_theme()
  theme_fivethirtyeight()
ggsave("imgs/films_by_year.png", width = 8, height = 5)

The first Kung-fu Shaw Brothers film in this data set is Temple of the Red Lotus from 1965. From the reviews, it sounds like it was a bit rough around the edges - but thats about what you would expect from this burgeoning genre.

The studio hits its stride in the early 70’s, with a lull in the mid 70’s and another spike in the late 70’s, early 80’s. Keep in mind that even during the lull, the studio is still putting out 10 or more Kung-fu movies most years.

Shaw Brothers Directorial Favorites

We have the director for each movie in our dataset, let’s look to see if there are any popular standouts.

by_director <- films %>% group_by(director) %>% summarise(n = n()) %>% arrange(-n)
  by_director %>% filter(n > 1) %>%
  ggplot(aes(x = fct_reorder(director, n), y = n)) + 
  geom_bar(stat = "identity") +
  coord_flip() +
  labs(title = 'Counts of Shaw Bros Films by Director') +
  theme_fivethirtyeight()
ggsave("imgs/director_count.png", width = 8, height = 6)

I’d say! Chang Cheh directed 67 or roughly 26% of all Shaw Brothers Kung-fu!

According to his Wikipedia page, he was known as the “The Godfather of Hong Kong cinema”, and rightly so - at least in terms of quantity.

Let’s pull out the top 5 directors, in terms of movie count, and see when they were most active.

# pull out just the top 5 directors
top_directors <- by_director %>% head(n = 5)
# filter films to those directed by these titans of kung-fu
films_top_director <- films %>% filter(director %in% top_directors$director)
#plot bar chart
films_top_director %>%
  ggplot(aes(x = year)) +
  geom_bar(aes(fill = director)) +
  labs(title = 'Shaw Bros Top Director Count by Year') + 
  theme_fivethirtyeight()

# Try Fill Position
films_top_director %>%
  ggplot(aes(x = year)) +
  geom_bar(aes(fill = director), position = "fill") +
  labs(title = 'Shaw Bros Top Director Count by Year') + 
  theme_fivethirtyeight()

That was with filtering to just the top directors. What happens when we put all of them in?

films_top_director_all <- films %>% mutate(director_label = ifelse(director %in% top_directors$director, director, 'Other'))
films_top_director_all %>%
  ggplot(aes(x = year)) +
  geom_bar(aes(fill = director_label)) +
  labs(title = 'Shaw Bros Director Count by Year', fill = '') + 
  theme_fivethirtyeight()
ggsave("imgs/top_directors_by_year.png", width = 8, height = 5)

films_top_director_all %>%
  ggplot(aes(x = year)) +
  geom_bar(aes(fill = director_label), position = 'fill') +
  labs(title = 'Shaw Bros Director Count by Year', fill = '') + 
  theme_fivethirtyeight() +
  scale_y_continuous(labels = percent)
ggsave("imgs/top_directors_by_year_fill.png", width = 8, height = 5)

Title Showdown

library(tidytext)

titles <- films %>% mutate(raw_title = title) %>% unnest_tokens(word, title)

titles_filter <- titles %>% anti_join(stop_words, by = "word")

by_word <- titles_filter %>% count(word, sort = TRUE)
by_word %>%
  filter(n > 3) %>% 
  ggplot(aes(x = fct_reorder(word, n), y = n)) +
  geom_bar(stat = 'identity') +
  coord_flip() + 
  labs(title = 'Top Words Used in Kung-Fu Titles') +
  theme_fivethirtyeight()
ggsave("imgs/top_words_in_titles.png",  width = 8, height = 6)

top_word <- by_word %>% head(n = 12)
films_top_word <- titles %>% filter(word %in% top_word$word)
films_top_word %>% 
  ggplot(aes(x = year)) +
  geom_bar() +
  labs(title = 'Titles with Most Common Words by Year') + 
  facet_wrap( ~ fct_relevel(word, top_word$word)) +
  scale_x_discrete(labels = function(x) { return(ifelse(as.numeric(x) %% 2, x, '')) }) +
  theme_fivethirtyeight() + 
  theme(axis.text.x = element_text(angle = 45, hjust = 1), legend.position="none")
ggsave("imgs/top_words_by_time.png", width = 8, height = 6)

Swordsman or Shaolin?

top2_word <- by_word %>% head(n = 2)
films_top2_word <- titles %>% filter(word %in% top2_word$word)
films_top2_word %>% 
  ggplot(aes(x = year, fill = fct_inorder(word))) +
  geom_bar() +
  labs(title = 'Swordsman vs Shaolin by Year', fill = '') + 
  #scale_x_discrete(labels = function(x) { return(ifelse(as.numeric(x) %% 2, x, '')) }) +
  theme_fivethirtyeight() 
  #theme(axis.text.x = element_text(angle = 45, hjust = 1))
ggsave("imgs/shaolin_swordsman.png", width = 8, height = 5)

Swordsman Titles

titles %>% filter(word == 'swordsman') %>% arrange(year) %>% select(raw_title, year)

swordsman success: https://en.wikipedia.org/wiki/One-Armed_Swordsman

It was the first of the new style of wuxia films emphasizing male anti-heroes, violent swordplay and heavy bloodletting. It was the first Hong Kong film to make HK$1 million at the local box office, propelling its star Jimmy Wang to super stardom.

Wuxia Films.

30 Essential Wuxia Films

The Chinese martial arts movie is generally split into two primary subgeneres: the kung fu film and the wuxia film. The kung fu film is newer and focuses primarily on hand-to-hand combat, it’s steeped in traditional fighting forms and there’s a general emphasis on the physical skill of the performer: special effects are generally disdained. Bruce Lee and Jackie Chan are its most famous practitioners and Lau Kar-leung its most important director.

Wuxia is a much older form, based ultimately in the long tradition of Chinese adventure literature, in classic novels such as The Water Margin or Journey to the West, or more contemporary works by authors like Louis Cha and Gu Long. Its heroes follow a very specific code of honor as they navigate the jianghu, an underworld of outlaws and bandits outside the normal streams of civilization.

From Wuxia Women Warriors

Less realistic than its cousin, the kung fu film, wuxia often includes gravity-defying stunts where legendary warriors fly through the air or punch holes straight through their enemies’ chests.

Shaolin Titles

titles %>% filter(word == 'shaolin') %>% arrange(year) %>% select(raw_title, year)

From History in the Shaw Brothers

These films, focused on the Shaolin Temple as a center for anti-Qing resistance, provide a dizzying metaphorical potential, with the Qing variously standing in for Western imperialists, the Japanese, the Nationalist Kuomingtang, the Communists, or even simply the Manchurians themselves, while the Buddhism of the monks allows for examining of the contradictions at the heart of traditional Chinese belief systems, between the imperatives of social justice and withdrawal from worldly concerns.

Actor Troupes and Groups

Even with my novice-level consuption of Shaw Brothers films, one thing I noticed early on was a lot of familiar faces that showed up in many of the movies. My assumption is just like directors, there are a number of actors that are heavily reused in these films. Let’s see if I am right!

First, just like directors, we can look at counts by actor.

by_actor <- cast %>% group_by(name) %>% summarise(n = n()) %>% arrange(-n)
by_actor %>% filter(n > 20) %>%
  ggplot(aes(x = fct_reorder(name, n), y = n)) +
  geom_bar(stat = "identity") +
  coord_flip() +
  labs(title = 'Counts of Shaw Bros Films by Actor') +
  theme_fivethirtyeight()

ggsave("imgs/actor_counts.png", width = 8, height = 6)

Wow! Ku Feng apparently appeared in 82 Kung-Fu movies. That’s a lot of Kung-Fu!

His Wikipedia page isn’t as impressed with this feat as I am, providing little information on this Martial Arts Maniac. Apparently his real name is Chan Sze-man, and his first film was in 1959, and apparently he is still acting. The HKMDB, or Hong Kong Movie Database, provides just a bit more info:

In 1965, Ku formally signed an acting contract with Shaw Brothers where he made around 100 films for them and became most notably known as one of their top character actors. He has worked with just about every top Hong Kong director in a variety of films.

Ok then, well props to you Ku.

Did most of the top actors’ carreers span multiple decades, or did actors come and go quickly? Let’s graph the top actor’s movie count by year.

top_actor <- by_actor %>% head(n = 16)
films_top_actor <- cast %>% filter(name %in% top_actor$name)

films_top_actor %>% 
  ggplot(aes(x = year)) +
  geom_bar() +
  labs(title = 'Top Actors Film Count by Year') + 
  facet_wrap( ~ fct_relevel(name, top_actor$name)) +
  # only label half of the years to make things a bit look cleaner
  scale_x_discrete(labels = function(x) { return(ifelse(as.numeric(x) %% 2, x, '')) }) +
  theme_fivethirtyeight() + 
  # angle label text
  theme(axis.text.x = element_text(angle = 45, hjust = 1), legend.position="none")
ggsave("imgs/top_actors_by_year.png", width = 6, height = 10)

Finding A Mob of Venoms

Venom Mob

Inspiration from Love Actually Analysis by David Robinson.

Create a matrix of actor co-occurance.

summary(by_actor$n)

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  1.000   1.000   2.000   4.572   4.000  82.000 

library(reshape2)
# filter actors not in many movies
min_movie_actors <- by_actor %>% filter(n > 5)
popular_cast <- cast %>% filter(name %in% min_movie_actors$name) 
cast_movie_matrix <- popular_cast %>%
  acast(name ~ title,  fun.aggregate = length)

Using name as value column: use value.var to override.

dim(cast_movie_matrix)

[1] 120 258

Rows are actors. Columns are movies.

Filter out movies with few co-occuring actors

cast_movie_df_filtered <- cast_movie_df %>% colSums(.)

norm <- cast_movie_matrix / rowSums(cast_movie_matrix)
hc_norm_cast <- hclust(dist(norm, method = "manhattan"))

Plot:

library(ggdendro)
ggdendrogram(hc_norm_cast, rotate = TRUE)

See ordering:

ordering <-hc_norm_cast$labels[hc_norm_cast$order]
ordering

  [1] "Wong Ching-Ho"         "Kara Hui"              "Gordon Liu Chia-Hui"   "Hsiao Ho"             
  [5] "Wilson Tong"           "Lau Kar-Wing"          "Liu Chia-Liang"        "Chin Ping"            
  [9] "Jimmy Wang Yu"         "Lisa Chiao Chiao"      "Tien Feng"             "Candy Wen Xue-Er"     
 [13] "Lung Tien-Hsiang"      "Philip Kwok Chun-Fung" "Chiang Sheng"          "Lu Feng"              
 [17] "Lo Meng"               "Sun Chien"             "Wong Lik"              "Yu Tai-Ping"          
 [21] "Chin Siu-Ho"           "Lam Chi-Tai"           "Lau Fong-Sai"          "Chiu Hung"            
 [25] "Wong Chung-Shun"       "Tung Li"               "Law Hon"               "Lee Pang-Fei"         
 [29] "Chin Han"              "Wang Ping"             "Ou-Yang Sha-Fei"       "Ivy Ling Po"          
 [33] "Lam Jing"              "Chiang Nan"            "Fung Ngai"             "Fung Hak-On"          
 [37] "Lo Dik"                "Chi Kuan-Chun"         "Leung Kar-Yan"         "Wang Han-Chen"        
 [41] "Choh Seung-Wan"        "Helen Poon Bing-Seung" "Lee Hoi-Sang"          "Phillip Ko Fei"       
 [45] "Wong Yu"               "Chen Kuan-Tai"         "Chiang Tao"            "Chan Sze-Kai"         
 [49] "Yeung Jing-Jing"       "Dang Wai-Ho"           "Jason Pai Piao"        "Kwan Fung"            
 [53] "Ng Hong-Sang"          "Yuen Tak"              "Lo Wei"                "Chiu Sam-Yin"         
 [57] "Hung Lau"              "Walter Tso Tat-Wah"    "Austin Wai"            "Wong Mei-Mei"         
 [61] "Keung Hon"             "Lam Fai-Wong"          "Alexander Fu Sheng"    "Johnny Wang Lung-Wei" 
 [65] "Bruce Tong Yim-Chaan"  "Dick Wei"              "Tang Ching"            "Wong Yung"            
 [69] "Cheng Pei-Pei"         "Lan Wei-Lieh"          "Cheng Lui"             "Wang Kuang-Yu"        
 [73] "Chan Sing"             "Lau Gong"              "Cliff Lok"             "Li Ching"             
 [77] "Shih Szu"              "Wai Wang"              "Chung Wa"              "Teresa Ha Ping"       
 [81] "Bolo Yeung"            "Tin Ching"             "Chen Ping"             "Danny Lee Sau-Yin"    
 [85] "Wong Chung"            "Lau Wing"              "Norman Chu"            "Ku Kuan-Chung"        
 [89] "Ching Li"              "Ngaai Fei"             "Yuen Bun"              "Yuen Wah"             
 [93] "Candice Yu On-On"      "Derek Yee Tung-Sing"   "Lau Wai-Ling"          "Ling Yun"             
 [97] "Tung Lam"              "Fan Mei-Sheng"         "Lily Ho Li-Li"         "Lily Li Li-Li"        
[101] "David Chiang"          "Ti Lung"               "Lee Wan-Chung"         "Chen Hung-Lieh"       
[105] "Fang Mian"             "Shu Pei-Pei"           "Tang Ti"               "Chang Yi"             
[109] "Goo Man-Chung"         "Shum Lo"               "Wu Ma"                 "Cheung Pooi-Saan"     
[113] "Dean Shek"             "Chan Shen"             "Wang Hsieh"            "Ku Feng"              
[117] "Lo Lieh"               "Elliot Ngok"           "Cheng Miu"             "Yeung Chi-Hing"       

ordered_films <- popular_cast %>% arrange(year) %>%
  mutate(film_index = as.numeric(fct_inorder(factor(title))), cast_name = factor(name, levels = ordering))
ordered_films %>% ggplot(aes(film_index, cast_name)) +
    geom_point() +
    geom_path(aes(group = film_index))

# http://stackoverflow.com/questions/13281303/creating-co-occurrence-matrix
cooccur <- cast_movie_matrix %*% t(cast_movie_matrix)
diag(cooccur) <- 0
heatmap(cooccur, symm = TRUE )

cooccur is matrix with rows and columns as actors. The cells for each actor combo indicate the number of movies they have appeared together in.

summary(rowSums(cooccur))

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   37.0    63.5    87.5   117.5   143.2   603.0 

summary(colSums(cooccur))

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   37.0    63.5    87.5   117.5   143.2   603.0 

summary(colSums(cooccur != 0))

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  16.00   34.75   45.00   47.55   54.00  110.00 

collab_counts <- as.data.frame(colSums(cooccur != 0))

library(igraph)
cooccur <- cast_movie_matrix %*% t(cast_movie_matrix)
#cooccur <- ifelse(cooccur < 4, 0, cooccur)
g <- graph.adjacency(cooccur, weighted = TRUE, mode = "undirected", diag = FALSE)
summary(E(g)$weight)

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   1.00    1.00    2.00    2.47    3.00   29.00 

summary(degree(g))

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  16.00   34.75   45.00   47.55   54.00  110.00 

summary(strength(g))

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   37.0    63.5    87.5   117.5   143.2   603.0 

library(igraph)
cooccur <- cast_movie_matrix %*% t(cast_movie_matrix)
#cooccur <- ifelse(cooccur < 4, 0, cooccur)
g <- graph.adjacency(cooccur, weighted = TRUE, mode = "undirected", diag = FALSE)
low_degree_v <- V(g)[degree(g) < 10] #identify those vertices part of less than three edges
g <- delete_vertices(g, low_degree_v) #exclude them from the graph
low_weight_e <- E(g)[E(g)$weight < 3]
g <- delete_edges(g, low_weight_e)
low_strength_v <- V(g)[strength(g) < 90]
g <- delete_vertices(g, low_strength_v) #exclude them from the graph
V(g)$betweenness <- strength(g)
plot(g, edge.width = E(g)$weight, 
     #layout=layout.fruchterman.reingold,
     layout=layout_with_fr,
     vertex.label.dist=0.5,
     #vertex.size = V(g)$betweenness,
     vertex.size = 3,
     vertex.color='steelblue',
     vertex.frame.color='white',        #the color of the border of the dots 
     vertex.label.color='black',        #the color of the name labels
     vertex.label.font=2,           #the font of the name labels
     vertex.label.cex=1,            #specifies the size of the font of the labels. can also be made to vary
     edge.color = hsv(0,0.2,0.5,alpha=0.2)
)

chiang_sheng <- V(g)[V(g)$name == "Chiang Sheng"]
chiang_sheng_neighbors <- neighbors(g, chiang_sheng)
neighbor_edges <- incident_edges(g, chiang_sheng_neighbors)
sub_g <- make_empty_graph(n = length(chiang_sheng_neighbors), directed = FALSE)
add_vertices(sub_g, chiang_sheng_neighbors)

IGRAPH U--- 12 0 -- 
+ edges:

#add_edges(sub_g, neighbor_edges)
E(g)[neighbor_edges]$color = hsv(0,0.2,0.5,alpha=0.5)

Error in structure(seq_along(x), names = names(x))[i] : 
  invalid subscript type 'list'

plot(g, edge.width = E(g)$weight, 
     #layout=layout.fruchterman.reingold,
     layout=layout_with_fr,
     vertex.label.dist=0.5,
     #vertex.size = V(g)$betweenness,
     vertex.size = 5,
     vertex.color=V(g)$color,
     vertex.frame.color='steelblue',        #the color of the border of the dots 
     vertex.label.color='black',        #the color of the name labels
     vertex.label.font=2,           #the font of the name labels
     vertex.label.cex=1,            #specifies the size of the font of the labels. can also be made to vary
     edge.color = hsv(0,0.2,0.5,alpha=0.2)
)

cooccur_df <- data.frame(cooccur, col.names = colnames(cooccur))
cooccur_df$name <- row.names(cooccur) 
cooccur_df <- cooccur_df %>% select(name, everything())
write_delim(cooccur_df, 'shaw_cooccurance.csv', delim = ';')

Other

R Markdown Notes

This is an R Markdown Notebook. When you execute code within the notebook, the results appear beneath the code.

Try executing this chunk by clicking the Run button within the chunk or by placing your cursor inside it and pressing Cmd+Shift+Enter.

Add a new chunk by clicking the Insert Chunk button on the toolbar or by pressing Cmd+Option+I.

When you save the notebook, an HTML file containing the code and output will be saved alongside it (click the Preview button or press Cmd+Shift+K to preview the HTML file).